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Overview of NBES Annual Report for IES 

In 1999 the National Research Council (NRC) published a report concluding that, “the 
complex world of education — unlike defense, health care, or industrial production — does 
not rest on a strong research base. Three years later, as the nation turned its attention to 
reforming federal education policy, the legislative branch and the executive branch 
agreed that the research base described in the NRC report was still woefully inadequate 
and needed more rigor and relevance. 

Congress, to its credit, responded by passing the Education Sciences Refonn Act of 2002 
(ESRA), establishing the Institute of Education Sciences (the Institute) and its advisory 
board, the National Board for Education Sciences (NBES). The Senate confirmed the 
members of the National Board for Education Sciences at the close of the 108 th Congress, 
and the Board held its first meeting in February 2005. ESRA mandates that the Board 
shall submit an annual report “ that assesses the effectiveness of the Institute in carrying 
out its priorities and mission, especially as they relate to carrying out scientifically valid 
research, conducting unbiased evaluations, and collecting and reporting accurate 
education statistics. ” 

For this report, the Board assessed the Institute’s progress in three areas: 1) the 
establishment of research priorities to guide its investments into the next decade; 2) the 
creation of a rigorous peer review process for grants applications and reports; and 3) the 
Institute’s efforts to ensure its work is useful to practitioners and policymakers. 

Here are the main findings in this report: 

• Research Priorities 

The Institute has developed clear research priorities that provide a strong, long-term 
guide for its work. 

• Plan for Addressing the Priorities 

The Institute has made great progress in funding projects that will fill important gaps 
in infonnation. 

• Standards and Policies for Peer Review 

The Institute has implemented standards and procedures for reviewing and approving 
grants and IES reports that are of the highest merit and are comparable to those of the 
National Science Foundation and the National Institutes of Health. 

• Actions Taken to Assure the Relevance of Its Work 

The Institute has initiated a number of activities to assure the relevance of its work for 
policy and practice. 

NBES believes that the vision Congress had when it created the Institute for Education 
Sciences is being realized, and the Board is privileged to serve during this important 
period of transformation. 
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I. Introduction 



In 1996 the National Research Council (NRC) began assessing the feasibility of a long- 
term research program focused on the topics that would be most critical to improving 
learning in the nation’s schools. When the panel’s findings were published in 1999, the 
NRC concluded that, “the complex world of education — unlike defense, health care, or 
industrial production — does not rest on a strong research base. In no other field are 
personal experience and ideology so frequently relied on to make policy choices, and in 
no other field is the research base so inadequate and little used” (. Improving Student 
Learning: A Strategic Plan for Education Research and Its Utilization, 1999). 

Those were tough words, but three years later, as the nation turned its attention to 
reforming federal education policy, the legislative branch and the executive branch 
agreed that the research base described in the NRC report was still woefully inadequate 
and needed more rigor and relevance. Congress, to its credit, responded by passing the 
Education Sciences Reform Act of 2002 (ESRA), establishing the Institute of Education 
Sciences (the Institute) and its advisory board, the National Board for Education Sciences 
(NBES). Through the work of the Senate Committee on Health, Education, Labor and 
Pensions and the House Committee on Education and the Workforce, as well the House 
and Senate Appropriation Committees, Congress created a new entity within the 
Education Department that would make significant strides in the quality and quantity of 
the research it sponsored. Free of partisan political influence, the Institute would report to 
Congress yearly on the condition of education in the United States. It would provide 
thorough and objective evaluations of federal programs such as Title I and Reading First. 
It would sponsor research that would be relevant and useful to educators and others. And 
it would become the place to go — the gold standard — for reliable, unbiased infonnation 
on what works in education — and what does not. 

Members of the National Board for Education Sciences were nominated by the President 
and confirmed by the Senate. The Board was confirmed by the Senate at the close of the 
108 th Congress, and held its first meeting in February 2005. To date, the Board has met 5 
times. NBES is composed of 14 researchers and practitioners who feel privileged to be a 
part of this groundbreaking attempt to improve the quality of education research. The 
ESRA mandates that the Board shall submit an annual report “that assesses the 
effectiveness of the Institute in carrying out its priorities and mission, especially as they 
relate to carrying out scientifically valid research, conducting unbiased evaluations, and 
collecting and reporting accurate education statistics. ” The Board is pleased to submit 
the attached report in the hopes that it will convey the extent of the progress the Institute 
has made as it transforms education into an evidence-based field in which decision- 
makers turn to rigorous data to inform policies and practices. 

From its inception, the Institute made it a first priority to develop policies and procedures 
that foster the highest scientific standards for the research it supports. The Board has 
played a role in institutionalizing the commitment to these standards by fulfilling its 
responsibility to formally approve the Director’s priorities and the procedures for peer 
review. To be thoughtful in its deliberations over the priorities and procedures, the Board 
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conducted its own review of the Institute’s work. As result, the Board is positioned to 
report on how the priorities were developed and to assess the plans to implement them. 
The Board also will provide information about how the peer review process was 
developed and how it is operating. 

II. Institute Research Priorities 



A. Background 

ESRA 2002 requires that the Director of the Institute propose to the Board long-tenn 
research priorities for the Institute, and that upon approval of such priorities, the Director 
make the Institute’s plan for addressing such priorities available for public comment. 

B. Development and Approval of the Priorities 

In July 2005, Dr. Grover “Russ” Whitehurst, Director of the Institute, developed a 
proposed set of priorities and published these in the Federal Register to solicit public 
comment. Over the summer, the Board prepared an analysis of the public comments, and 
shared it with Institute senior staff. Dr. Whitehurst considered the Board’s analysis and 
the public comments and modified his proposed draft. 

During the September 2005 Board meeting, following review and discussion of the 
revised statement of priorities, the Board approved the Institute priorities. Dr. Whitehurst 
summarized the overview, goals and targets of the final Institute research priorities in the 
following way (the entire text can be found in attachment A): 

Overview of Institute Research Priorities 

• By providing an independent, scientific base of evidence and promoting and 
enabling its use, the Institute aims to further the transformation of education 
into an evidence-based field, and thereby enable the nation to educate all of its 
students effectively. 

• In pursuit of its goals, the Institute will support research, conduct evaluations, 
and compile statistics in education that conform to rigorous scientific 
standards, and will disseminate and promote the use of research in ways that 
are objective, free of bias in their interpretation, and readily accessible. 

Goals 

• To develop or identify a substantial number of programs, practices, policies, 
and approaches that enhance academic achievement and that can be widely 
deployed; 

• To identify what does not work and what is problematic or inefficient, and 
thereby encourage innovation and further research; 
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• To gain fundamental understanding of the processes that underlie variations in 
the effectiveness of education programs, practices, policies, and approaches; 
and 

• To develop delivery systems for the results of education research that will be 
routinely used by policymakers, educators, and the general public when 
making education decisions. 

What and Who 

The Institute's over-arching priority is to support research that contributes to 
improved academic achievement for all students, particularly for those students 
whose education prospects are hindered by inadequate education services and 
conditions associated with poverty, race/ethnicity, limited English proficiency, 
disability, and family circumstance. 

C. Board’s Assessment of the Effectiveness of the Institute in Developing Its 
Priorities 

The Institute did an excellent job developing its research priorities. The draft prepared for 
public comment was a sound document, reflecting the Institute’s current work. It was 
more focused than many of the previously developed priority statements. That said, the 
public and Board comments concentrated on important new areas such as special 
education and the use of research. The Director and the Institute were appropriately 
responsive to the comments, and the final draft provides a strong long-term guide. 

D. Plan to Address the Priorities 

The priorities are a strategic statement for goals and areas of intent but by design they are 
not a strategic plan. Therefore, once the Board had approved the priorities, the Director 
set about developing a plan to address the priorities. Logically, the research that the 
Institute has funded to date is consistent with the Director’s priorities. However, because 
these priorities are intended to guide work over the long term, this plan shows how the 
Institute will ensure that future work is strategically driven and that gaps are identified 
and filled. The Institute’s primary mechanism for acting on the priorities is through 
research competitions to which researchers and research organizations can apply for 
funding. These competitions are organized by topic (e.g., reading and writing), and by 
center within the Institute (e.g., the National Center for Special Education Research). The 
plan explains how this mechanism will be used to oversee adherence to the priorities and 
take corrective action as necessary. 

The plan has three components: opportunities for researchers, mix of grant applications, 
yield of grants. Each component involves an assessment of current activities to detennine 
if action is necessary, and where necessary, to create new research activities or modify 
existing activities in order for the Institute’s portfolio of activities to cover the priorities. 
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• Opportunities for Researchers 

The first component involves assessing whether the Institute is providing opportunities 
for researchers to obtain funding for work on each of the topics identified in the priorities. 
For example, is the Institute providing funding opportunities for research on the effects of 
curriculum and instruction on outcomes in mathematics and science outcomes for 
students in grade K-12? If such an assessment revealed a gap in the topics being funded, 
the corrective action would be to create new funding opportunities. 

The number of topics covered in the competitions has increased each year, and numbered 
20 for the 2006 fiscal year. As an example of the topic areas, the table below presents the 
number of grants awarded by topic area in 2006. 1 

Topic Areas 2006 Number of 

Grants 



Statewide Longitudinal Data Systems 14 

Cognition and Student Learning 10 

Reading and Writing 7 

Serious Behavior Disorders 7 

Early Intervention 7 

Mathematics 6 

Research and Development Centers 4 

Post-Doctoral Fellowships 4 

NAEP Secondary Analysis 3 

Language Development 3 

Finance, Leadership, and Management 2 

Teacher Quality 2 



69 



• Mix of Grant Applications 

The second component involves assessing whether the mix of grant applications within 
each topic is appropriate to the Institute’s goals of detennining what works and does not 
work, and understanding the processes that underlie variations in program effectiveness. 
For example, with respect to the effects of curriculum and instruction on mathematics and 
science outcomes, is there sufficient upstream work to generate a new generation of 
programs and practices? Is there sufficient downstream work in moving interventions to 
scale and evaluating their effectiveness? When the mix of work within a research 
program is unbalanced, the corrective actions include: clarifying funding announcements, 
increasing capacity in the research community, and enhancing incentives to pursue 
particular categories of research. 

• Yield of grants 



1 The Institute has not yet completed the peer review and award process for applications submitted on topics not 
covered by topical funding announcements. 
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The third component involves assessing whether the yield of grants within each topic is 
advancing the Institute’s goals, particularly the goal of developing and identifying 
programs and practices that are effective in enhancing academic achievement. For 
example, are grants on the effects of curriculum and instruction on mathematics and 
science outcomes yielding promising findings that can be field-tested? Are field tests of 
programs and practices at scale yielding positive effects? When the yield from a research 
program is insufficient, the Institute can redirect investments and use different forms of 
funding — such as contracts and cooperative agreements — to jumpstart and actively shape 
the research portfolio. (The full research plan that was submitted to the Federal Register 
for public comment can be found in Attachment B.) 



E. Board Assessment of the Institute’s Plan for Addressing the Priorities 

Because the Institute is relatively young, it has focused on developing new opportunities 
for researchers. In reviewing the plan for addressing the priorities and the grants that 
were funded in the last year, the Board concludes that the Institute has made great 
progress in filling the gaps. 

The second component, the mix of types of research within research programs, will be 
addressed for those research topics that have been through at least two cycles of funding. 
Addressing the third component, yield, is just beginning to be possible at this point. The 
Board will be reviewing the grants funded by subject each year to assess whether the 
corrective actions outlined in the plan have been implemented as needed. 



III. Scientifically Valid Research, Unbiased Evaluations 
and Accurate Education Statistics 



A. Background 

ESRA requires the Director of the Institute “to establish necessary procedures for 
technical and scientific peer review of the activities of the Institute” to assure that 
“scientifically based research standards’’ are applied to the work of the Institute, 
including the funding of grant applications and the products that are published by the 
Institute. To that end, the Standards and Review Office was created and staffed within the 
Institute’s Office of the Deputy Director for Science. The Standards and Review Office 
developed, implemented, and refined the peer review procedures beginning shortly after 
the enactment of ESRA. The review procedures apply to all reports supported by the 
Institute, as well as all grant proposals submitted for funding. 

B. Peer Review of Products 

ESRA requires that “all research, statistics, and evaluation reports conducted by, or 
supported through, the Institute shall be subjected to rigorous peer review before being 
published or otherwise made available to the public.” In addition, the Act requires that 
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Institute products be “objective, secular, neutral, and non-ideological and are free of 
partisan political influence and racial, cultural, gender, or regional bias.” Locating the 
Standards and Review Office within the Office of the Deputy Director for Science, 
independent of any of the Institute’s four operating Centers, allows Institute staff 
responsible for approval and scientific peer review of products (action editors) to 
maintain an independent, objective point of view. Action editors are senior staff members 
with several years of experience conducting research, publishing and carrying out 
scientific reviews. Center staff responsible for the oversight of research projects, research 
contractors, and report authoring are necessarily exposed to influences and relationships 
that could easily result in the appearance of conflict of interest in a review process. This 
carefully engineered organizational independence allows the Institute’s products to 
undergo objective, anns-length peer review, setting a standard for rigor, and placing the 
Institute on par with the nation’s other premier federal science agencies. 

Centers submit a report to the Standards and Review Office after they have conducted 
their own Center-level review, and the relevant Commissioner has approved the report 
for submission. The relevant Center Commissioner recommends to the Deputy Director 
for Science whether a report should receive external or internal scientific review, or 
should be exempt from review. Reports that present new analyses of data are sent to 
external scientists for peer review. All reports with limited descriptive data analyses are 
reviewed internally by Standards and Review staff. 

For an external scientific review, the action editors identify potential peer reviewers, 
locating researchers who have published in top research journals in the relevant content 
area, have used similar methodological approaches, and have substantial experience 
conducting studies similar to the one presented in the report. The Deputy Director for 
Science reviews names, publication lists, and examples of published work of potential 
reviewers, and approves the recruitment of specific reviewers. Action editors recruit at 
least two approved external reviewers for each report. The external reviewers are asked to 
focus on the significance of the work for the field of education (the research itself, and 
the product at-hand); the technical quality of the research design, data collection, and data 
analyses; the appropriateness of the conclusions; and the clarity of the presentation. They 
are also asked to assure that any language in the report that advances causal claims is 
supported by the research methods and analyses described in the report. 

Standards and Review Office action editors conduct their own review of each report 
simultaneously with the external reviewers, and then write a disposition memorandum 
synthesizing the action editor’s review and those of the external reviewers. In their 
reviews, action editors focus on issues of technical quality, and are also responsible for 
ensuring that the reports are neutral and objective, and do not contain policy implication 
statements or recommendations, or statements of advocacy for particular positions, 
programs, or policies. The disposition memorandum indicates whether or not the report 
has been approved for publication, or is in need of revision. 

For an internal scientific review, the disposition memorandum is based on the action 
editor’s review of the report. For both internal and external reviews, the disposition 
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memorandum is reviewed and approved by the Deputy Director for Science, and then 
sent to the Center responsible for the report. Standards and Review action editors, in 
consultation with the Deputy Director for Science, are responsible for reviewing and 
approving revisions made to the report and for recommending final approval of the report 
for publication by the Institute. 

The Institute understands the importance of a timely review process. In 2005, it devoted 
an average of 24.9 working days up to the disposition memo (representing a 12% 
decrease from the previous year); 29.6 days in the Standards and Review Office (a 23% 
decrease from 2004); and 60.7 total working days until final approval (35% decrease 
from 2004). The Standards and Review Office is committed to continuing to address a 
timely review process. 

Implementation of the new peer review process has resulted in substantial changes to 
some Institute products. As with the review process used by top research journals, reports 
sometimes require more than one round of revisions prior to receiving approval for 
publication from the Standards and Review Office. Sometimes these revisions involve 
issues of technical quality, and sometimes the revisions are focused on ensuring that the 
work is presented in a clear, objective, neutral, and unbiased manner. As an example, all 
program evaluation reports approved by the new review process will clearly lay out the 
purpose of the report, the study context, research design, data collection procedures, 
outcome measures, and data analyses; report the findings in a neutral manner; and leave 
implications for policy and practice to the reader. 

The scientific review process for reviewing products represents a departure from the way 
in which Department research, statistics, and evaluation products were handled prior to 
passage of ESRA. While products were subject to various kinds of review prior to 
publication or public release, there was no systematic, independent scientific peer review 
of research, statistics, and evaluation products, and there were no requirements that such 
products be neutral and objective. The new process is intended to mirror the scientific 
peer review process used by top research journals for the acceptance of research articles 
for publication. 

The Institute’s peer review process for reports was initiated in early 2004. In 2004, the 
Standards and Review Office reviewed 44 reports, and in 2005, 96 reports. 

C. Peer Review of Grant Applications 

Under ESRA, activities of the Institute that are carried out through grants, contracts, or 
cooperative agreements, at a minimum, shall be awarded on a competitive basis and, 
when practical, through a process of peer review. Further, the Director is required to 
establish a peer review procedure (involving highly qualified individuals with an in-depth 
knowledge of the subject to be investigated) for reviewing and evaluating all applications 
for grants and cooperative agreements that exceed $100,000. 
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The Standards and Review Office is responsible for implementing the scientific peer 
review of grant applications. As with the peer review of reports, a key provision of the 
grant application peer review system is intended to put distance between the program 
officers and administrators within the Institute who administer grant programs, work with 
grantees, and disseminate the results of research, and those who are responsible for the 
peer review of applications for funding under those grant programs. 

In FY2002, the Institute established a new system for the scientific review of grant 
applications that is similar to the process of grant application peer review at the National 
Institutes of Health. Standards and Review Office staff identify and recruit highly 
qualified reviewers primarily on the basis of the quality of the research they have 
conducted and published in scientific peer-reviewed journals and the degree to which 
they are in-depth experts in the relevant research methods and subject matter. 

These reviewers are assigned to various panels designated to review applications for 
similar research topics. Reviewers are asked to consider the significance of the work for 
the field of education, the quality of the research plan, the quality of the personnel, and 
the adequacy of the available resources for the project. Two to three primary reviewers 
provide independent narrative reviews and initial rating scores for each of the four 
criteria above, as well as an initial overall quality rating for their assigned application. 
Based on the initial overall scores provided by the primary reviewers, Standards and 
Review staff prepare a preliminary rank order of the applications assigned to each review 
panel, and based on this rank ordering, approximately the top 25 applications are 
identified for discussion by the full panels at the review panel meetings. 

In FY2003, the Institute created an entirely electronic application submission and review 
process. This electronic system allows applicants to submit letters of intent and grant 
applications on-line. In addition, the electronic system also allows reviewers to access 
applications assigned to their panel, submit their reviews of applications, view 
preliminary scores and reviews submitted by other reviewers (after submitting their own 
reviews), and revise their own narrative comments during the panel review meeting. The 
electronic system allows Institute staff to closely monitor the progress of the review 
process, and to quickly calculate preliminary scientific merit scores that are used to triage 
the top ranked applications for consideration by the full panels at the panel review 
meetings. 

Between FY2002 and FY2005, single session review panels were constituted as needed 
for each round of review, and each review panel considered one grant competition 
(topic). In FY2006, the Institute created 5 standing review panels to which panel 
members may be appointed for multiple, consecutive review sessions, to complement the 
use of single session review panels as appropriate. The standing panels are composed of 
principal panel members who serve staggered, 3-year terms, rotating panel members who 
serve for one review session, and ad hoc members who serve for one review session to 
review a subset of applications using specialized expertise. 

The following table summarizes the growth of the review process over the last five years. 
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Year 


Number of 
Applications 


Number of 
Competitions 


Number of 
Reviewers 


Number of 
Panels 


FY2002 


226 


4 


73 


4 


FY2003 


479 


7 


121 


7 


FY2004 


600 


8 


145 


8 


FY2005 


696 


11 


162 


11 


FY2006* 


703 




185 


5 standing 
5 single session 



* In 2005, the National Center for Special Education Research (NCSER) was established, and the 
Standards and Review Office became responsible for the peer review of grant applications from this Center 
starting in FY2006. 

The new peer review system for grants has been in place for sufficient time to consider its 
yield. Between FY2001 and FY2004, to examine whether the Institute was fulfilling its 
goal to fund rigorous research, a number of distinguished researchers were asked to rate 
the overall quality of newly funded research projects. The percent of newly funded 
research projects rated as being of high quality from FY2001 to FY2004 was 36%, 50%, 
70%, and 70%, respectively. 

The review processes and expectations for higher quality proposals have improved the 
quality of the responses from the field. In each of the fiscal years 2002 through 2004, the 
Institute funded about 8% of the grant applications received. In FY2005, the Institute 
funded about 12%. 

In addition, the Institute is also keeping track of the percentage of new research proposals 
funded by the Institute’s National Center for Education Research (NCER) that receive an 
average score of excellent or higher from a panel of independent reviewers. From 
FY2003 to FY2005 the percentage of proposals that received this score were 88%, 97% 
and 100%, respectively. 



D. Board Assessment and Approval of Peer Review Processes 

While the process for scientifically reviewing grant proposals was first implemented in 
2002 and the Institute’s peer review process for reports was initiated in early 2004, the 
Board did not formally approve these procedures until its September 2005 meeting. This 
time lag allowed the Board to review extensive documentation of these processes and to 
assess not only the quality of their design but also the fidelity of their implementation 
prior to granting fonnal approval. Board members received grant reviews and multiple 
iterations of report reviews. Members also attended panel meetings and thus were able to 
assess the quality of the deliberations. 

Many of the Board members are researchers who have received funding from the 
National Science Foundation (NSF) and/or the National Institutes of Health (NIH) and 
are familiar with the processes of these federal research agencies. They were impressed 
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with what the Institute had put in place and were able to validate that these processes 
would assure quality, objectivity, validity, and integrity in scientific publications. 

In addition, ex-officio members Arden Bement (NSF) and Duane Alexander (NICHD) 
publicly acknowledged that the approved procedures were of the highest merit and 
comparable to those of their agencies. While the processes are exemplary, the Board and 
Director realize that the key to strong systems is the selection of strong panel members. 
Members made several recommendations related to panel composition and asked that the 
Director institute quality assurance procedures to insure that these well-developed 
processes continue to be implemented with fidelity. The Director has committed to 
ongoing monitoring and reporting of the quality and compositions of the panels. 

IV. Making the Institute More Relevant to Policymakers and Practitioners 

Over the last six months, the Director and Institute senior staff have been considering 
ways to ensure that the Institute’s research better infonn the difficult questions that 
education policy-makers and practitioners are facing. In order to be more responsive to 
these constituencies and improve its dissemination mechanisms, the Institute has 
undertaken the following efforts: 

• Regional Educational Labs 

In the re-competition for 10 Regional Educational Labs, the Institute has added a 
requirement that they be able to conduct fast response applied research and 
development projects for the policymakers and practitioners in their region. 

• National Research and Development Centers 

Similarly, the Institute has established cooperative agreements with its 10 
research and development centers to enable quick turnaround studies to inform 
pressing policy concerns. 

• Practice Guides 

Practice Guides address practitioners’ current needs for coherent guidance based 
on evidence. The guides will be designed to incorporate the best available 
evidence, allowing for evidence below the standards for scientifically-based 
research in NCLB and the Institute statute if such evidence is the best available. 
Practice Guides will clearly label the quality of evidence supporting particular 
recommendations. The Practice Guides will include both summary 
recommendations and a supporting narrative. 

• Redesign of What Works Clearinghouse (WWC) 

The WWC is currently one of the Institute’s most significant dissemination 
mechanisms. However, it is still somewhat complex for practitioners to interpret 
the findings from the WWC. The redesign will provide practitioners with concise 
documents that summarize whether the evidence supporting particular programs 
and practices is positive, indeterminate, or negative. 
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The WWC is also changing a threshold for including strong designs for which 
sample size limitations have previously precluded inclusion in the WWC. This 
will increase the number of research studies in the database. 



• Statewide longitudinal databases 

The Institute is providing technical assistance to states that have received a grant 
to design, develop and implement statewide longitudinal data systems to 
efficiently and accurately manage, analyze, disaggregate, and use individual 
student data. 

• Collaborations 

The Institute is embarking on numerous collaborative activities with the Council 
of the Great City Schools, the Council of Chief State School Officers (CCSSO) 
and the National Conference of State Legislatures (NCSL). The Institute will be 
establishing a task force on urban education, composed of superintendents and 
researchers. The goal is for researchers to start addressing the questions that are 
important to urban school districts and for districts to buy into the value of 
conducting rigorous research on their policy initiatives. 



Board’s Assessment of the Institute’s Actions Taken to Assure the Relevance of Its 
Work 

While much of the Institute’s focus to date has been on assuring the rigor of the work it 
supports, the Board is very pleased that the Director and staff are conducting a number of 
activities to assure the relevance of the work for policy and practice. The items described 
above are an excellent start and the Board looks forward to additional examples and to 
the execution and assessment of the current plans. 

V. Upcoming Activities 

As this report makes clear, to date the Board has emphasized its statutory requirements to 
approve the Director’s priorities and the Institute’s peer review procedures. In the coming 
year, the Board will continue to monitor the processes that ensure the rigor of the 
Institute’s work, while supporting an ongoing dialog with the field to assure its relevance. 
In addition, the Board will be discussing a variety of issues that may lead to 
recommendations for congressional actions in the following areas: 

• Demonstration authority for evaluation of components of Elementary and 
Secondary Education Act (ESEA) that require compliance so that participating 
schools and districts could vary practices from those required under law in the 
context of evaluating the effects of those variations 

• Requirement for all congressionally mandated or authorized evaluations of 
Department of Education programs to be carried out by the Institute 
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• Ways of meeting the privacy requirements of Family Educational Rights and 
Privacy Act that allow independent research using data collected by local and 
state education agencies pursuant to ESEA 

• Fungibility of the .5% evaluation set aside for evaluation of small programs 

• Requirement of longitudinal data for reporting of graduation and other 
persistence outcomes 

• Rationale and funding levels for congressionally mandated national research 
and development centers 

• Reauthorization of ESRA 



VI. Recommendation Related to Strengthening Institute 

Overall, we have one overarching recommendation: Since the Institute has and continues 
to meet the highest standards for promoting rigorous and relevant research, the National 
Board for Education Sciences recommends that Congress and the Administration support 
the Institute’s role as the primary agency for federal education research. 
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Attachment A 



Priorities for the Institute of Education Sciences as Approved 
by The National Board for Education Sciences 
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Priorities for the Institute of Education Sciences as approved by The National Board 

for Education Sciences 



The National Board for Education Sciences has approved research priorities for the 
Institute of Education Sciences (IES). Proposed by IES Director Grover (Russ) 

Whitehurst under terms of the Education Science Reform Act of 2002, the priorities were 
submitted for public comment this summer and approved at the National Board's meeting 
September 6-7 in Washington, D.C. The text follows: 

The long-term goals associated with the Institute's priorities are fourfold: First, to develop 
or identify a substantial number of programs, practices, policies, and approaches that 
enhance academic achievement and that can be widely deployed; second, to identify what 
does not work and what is problematic or inefficient, and thereby encourage innovation 
and further research; third, to gain fundamental understanding of the processes that 
underlie variations in the effectiveness of education programs, practices, policies, and 
approaches; and fourth, to develop delivery systems for the results of education research 
that will be routinely used by policymakers, educators, and the general public when 
making education decisions. By providing an independent, scientific base of evidence 
and promoting and enabling its use, the Institute aims to further the transformation of 
education into an evidence-based field, and thereby enable the nation to educate all of its 
students effectively. 

In pursuit of its goals, the Institute will support research, conduct evaluations, and 
compile statistics in education that conform to rigorous scientific standards, and will 
disseminate and promote the use of research in ways that are objective, free of bias in 
their interpretation, and readily accessible. 

The Institute's over-arching priority is research that contributes to improved academic 
achievement for all students, and particularly for those whose education prospects are 
hindered by inadequate education services and conditions associated with poverty, 
race/ethnicity, limited English proficiency, disability, and family circumstance. 

With academic achievement as the major priority, the Institute will focus on outcomes 
that differ by periods of education. In the infancy and preschool period, the outcomes of 
interest will be those that enhance readiness for schooling, for example, language skills, 
and for infants and toddlers with disabilities, developmental outcomes. In kindergarten 
through 12th grade, the core academic outcomes of reading and writing (including 
reading and writing in the disciplines), mathematics, and science will be emphasized, as 
will the behaviors and social skills that support learning in school and successful 
transitions to employment, independent living, and post-secondary education. At the 
post-secondary level, the focus will be on enrollment in and completion of programs that 
prepare students for successful careers and lives. The same outcomes are emphasized for 
students with disabilities across each of these periods, and include the functional 
outcomes that improve educational and transitional results. The acquisition of basic skills 
by adults with low levels of education is also a priority. 
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In conducting research on academic outcomes, the Institute will concentrate on 
conditions within the control of the education system, with the aim of identifying, 
developing, and validating effective education programs, practices, policies, and 
approaches as well as understanding the factors that influence variation in their 
effectiveness such as implementation. Conditions that are of highest priority to the 
Institute are in the areas of curriculum instruction, assessment (including the 
identification of students with disabilities), the quality of the education workforce, and 
the systems and policies that affect these conditions and their interrelationships (for 
example, accountability systems, delivery mechanisms including technology, and policies 
that support the ability of parents to improve educational results for their children through 
such means as choice of education services and provision of school-related learning 
opportunities in the home). 

The successful pursuit of the Institute's goals and priorities requires increased capacity to 
produce and use rigorous education research. To that end, the Institute's priorities include 
support of post-doctoral training and interdisciplinary doctoral training in the education 
sciences, development and refinement of education research methods, and expansion and 
use for research of longitudinal databases that link student data to information on the 
conditions that affect outcomes, such as curriculum. To assure increased capacity to use 
and apply the results of research, the Institute will support systematic reviews of 
evidence, enhanced access to findings through advanced information systems, and 
outreach to parents, educators, students, policymakers, and the general public. 

These are not exclusive or absolute priorities: To the extent that resources pennit and the 
Institute's priorities are being adequately addressed, the Institute may address other 
important education issues. 
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Attachment B 



The Institute for Education Sciences Plan for Addressing its 

Research Priorities 
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The Institute of Education Sciences’ Plan for Addressing its Research Priorities 

The Education Sciences Reform Act of 2002 requires that the Director of the Institute of 
Education Sciences (Institute) propose to the National Board for Education Sciences 
(Board) long-tenn research priorities for the Institute, and that upon approval of such 
priorities, the Director make the Institute’s plan for addressing such priorities available for 
public comment. The Board has approved the Director’s priorities. This document 
describes the Institute’s plan for addressing those priorities. 

Review of Priorities 

The Institute’s research priorities (http ://ies. ed. gov/ director/board/priorities. asp) are driven 
by the goals of developing and identifying programs and practices that are effective in 
enhancing academic achievement, identifying existing programs and practices that do not 
work or are inefficient, gaining understanding of the processes that underlie variations in 
the effectiveness of programs and practices, and delivering results of education research to 
consumers in ways that will support the use of research evidence when making education 
decisions. 

These goals are to be pursued by research activities that focus on the effects on specific 
student outcomes of four conditions: curriculum and instruction, assessment, workforce 
quality, and the systems and policies under which the first three conditions operate. The 
specific student outcomes of interest vary by age of the student. They are: 

• Infancy and preschool 

o readiness for schooling 

o developmental outcomes for infants and toddlers with disabilities 

• Kindergarten through 12th grade 

o reading and writing 
o mathematics and science 

o behaviors and social skills that support learning in school and successful 
transitions, and additional functional outcomes for students with disabilities 

• Post-secondary level 

o enrollment in and completion of programs that prepare students for 
successful careers and lives 
o acquisition of basic skills by adults 

Mechanism for Acting on the Priorities 

The principal means by which the Institute can act on its research priorities is through 
research competitions to which researchers and research organizations can apply for 
funding. These competitions are organized by topic, e.g., reading and writing, and by 
center within the Institute, e.g., the National Center for Special Education Research. 
Annual research competitions have been held since the Institute was established in 2002. 
The number of topics covered in the competitions has increased each year, and numbered 
20 for the 2006 fiscal year. 
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The Plan for Addressing Priorities 



The Institute’s plan for addressing its research priorities has three elements, each involving 
an assessment of current activities to determine if new research activities should be created 
or existing activities should be modified in order for the Institute’s portfolio of activities to 
cover the priorities. 

The first element involves assessing whether the Institute is providing opportunities for 
researchers to obtain funding for work on each of the topics identified in the priorities 
(where the topics are defined as the intersect of the conditions of influence, outcomes, and 
age periods listed in the previous section). Is the Institute, for example, providing funding 
opportunities for research on the effects of curriculum and instruction on outcomes in 
mathematics and science during the k-12 period? The action associated with this 
assessment is to create new funding opportunities when gaps in coverage are identified. 

The second element involves assessing whether the mix of grant applications within each 
topic is appropriate to the Institute’s goals of determining what works, what doesn’t, and 
understanding the processes that underlie variations in program effectiveness. For 
example, with respect to the effects of curriculum and instruction on mathematics and 
science outcomes, is there sufficient upstream work to provide a new generation of 
programs and practices? Is there sufficient downstream work in moving interventions to 
scale and evaluating their effectiveness? A variety of actions can be taken when the mix 
of work within a research program is unbalanced. These include clarifying funding 
announcements, increasing capacity in the research community, and enhancing the 
incentives to pursue particular categories of research. 

The third element involves assessing whether the yield of grants within each topic is 
advancing the Institute’s goals, particularly the goal of developing and identifying 
programs and practices that are effective in enhancing academic achievement. For 
example, are grants on the effects of curriculum and instruction on mathematics and 
science outcomes yielding promising findings that can be field-tested? Are field tests of 
programs and practices at scale yielding positive effects? A variety of actions can be 
taken when the yield from a research program is insufficient. These include redirecting 
investments and using different forms of funding such as contracts and cooperative 
agreements to jumpstart and actively shape the research portfolio. 

The Institute, established in November of 2002, is only a little over 3 years old. Its grants 
typically run for terms of 4 years. At this stage of the Institute’s life the emphasis has to 
be on the first element of the plan, assuring that research opportunities exist for each of the 
Institute’s priorities. The second element, the mix of types of research within research 
programs, can be addressed for those research topics that have been through at least two 
cycles of funding. Addressing the third element, yield, is just beginning to be possible. 
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Do research opportunities fit the priorities? 

The Director’s research priorities were published in the fall of 2005. Those priorities built 
on research competitions that were already in place. In that context, it should not be 
surprising that most of the conditions and outcomes identified in the priorities are 
presently being covered. 

Kindergarten through 12 th grade. Each of the priorities is being covered for the k-12 
period. The effects on student outcomes of curriculum and instruction, assessment, and 
workforce quality are covered by the Institute’s research grants in the following areas: 

• reading and writing 

• reading, writing, and language development (special education) 

• teacher quality in reading and writing 

• quality of teachers and other service providers (special education) 

• mathematics and science 

• mathematics and science (special education) 

• teacher quality in math and science 

• cognition and student learning 

• individualized education programs and individualized family service plans (special 
education) 

• response to instruction (special education) 

• high school reform 

• interventions for struggling adolescent and adult readers 

• socialization and character development 

• serious behavior disorders (special education) 

• secondary and transition services (special education) 

• autism spectrum disorders (special education) 

The effects of systems and policies in kindergarten through 12 th grade are covered by the 
following research programs and national research and development centers: 

• education leadership 

• education policy, finance, and management 

• assessment for accountability (special education) 

• national assessment of educational progress secondary analysis 

• statewide, longitudinal data systems 

• national research and development centers on: 

o rural education 
o data-driven reform 

o school choice, competition, and achievement 
o evaluation, standards, and student testing 
o English language learners 
o performance-based incentives 
o analysis of longitudinal data 
o gifted and talented education 
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Infancy and preschool. Many, but not all, of the priorities are covered during the infancy 
and preschool period by existing research programs. Each of the teacher quality and 
subject matter topics listed in the previous section on k-12 education includes the pre-k 
period. Additional research programs that focus exclusively on infancy and preschool 
include: 

• early intervention, early childhood special education, and assessment for young 
children with disabilities (special education) 

• national research and development center on early childhood development and 
education 

• preschool curriculum evaluation 

A notable gap in coverage is special education research on infants and toddlers with 
disabilities. Accordingly, the Institute has just launched the early intervention, early 
childhood special education and assessment for young children with disabilities program, 
listed above, which covers children from birth through age five. In addition, the Institute’s 
new research program on autism spectrum disorders, listed in the previous section on k-12 
education, includes the infancy and preschool period. 

Postsecondary. The Institute has two research programs that focus exclusively on 
postsecondary education: 

• national research and development center on postsecondary education and training 

• postsecondary education 

In addition, a number of the research programs listed in the previous section on k-12 
education expressly include coverage of the postsecondary period. Specifically, 
interventions for struggling adolescent and adult readers, mathematics and science, teacher 
quality in reading and writing, and teacher quality in mathematics and science, include 
teaching basic skills to adults through adult and vocational education programs and 
developmental/bridge programs designed to help under-prepared students acquire the 
skills to succeed in college. The cognition and student learning program includes research 
on adult education and tertiary education. And the program in secondary and transition 
services includes the transition from secondary to postsecondary settings for students with 
disabilities. 

However, some areas of research identified by the priorities have been underrepresented in 
existing research programs. These include research on the effects of different forms of 
student aid on enrollment and persistence, the development and evaluation of remedial 
programs and retention programs for students who enter college underprepared, and the 
development and validation of assessments that capture the value added by baccalaureate 
programs in preparing students for careers after graduation. The postsecondary education 
research program, listed above, has just been launched to address these issues. 

Is the mix of research appropriate? 

The Institute funds research in seven categories: (1) Identification - identify existing 
programs, practices, and policies that are differentially associated with student outcomes 
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and the factors that may mediate or moderate the effects of these programs, practices, and 
policies; (2) Development - develop programs, practices, and policies that are potentially 
effective for improving outcomes; (3) Efficacy - establish the efficacy of fully developed 
programs, practices, or policies that either have evidence of potential efficacy or are 
widely used but have not been rigorously evaluated; (4) Scale-up - provide evidence on 
the effectiveness of programs, practices, and policies implemented at scale; (5) 

Assessment and Tools - develop or validate data and measurement systems and tools; (6) 
Training - increase the capacity to conduct rigorous and relevant research through pre- 
doctoral and postdoctoral training of education scientists; and (7) Centers - carry out a 
range of research and national leadership on topics specified by Congress. 

Categories 1 through 4 are a logical and progressive ordering of research activities towards 
the goal of developing and identifying programs and practices that are effective in 
enhancing academic achievement. Category 4, scale-up research, is the logical culmination 
of research activities that identify and develop promising approaches (categories 1 and 2) 
and establish their efficacy in small-scale field tests (category 3). While there is no 
formula for detennining the appropriate mix of research across these categories, the 
Institute wants to see a distribution that has the shape of a triangle, with the base 
consisting of identification and development activities, the second level representing 
small-scale field tests, and the apex representing evaluations of programs and practices at 
scale. Table 1 is a tabulation of applications in each of categories 1 through 4 as a 
percentage of the total number of applications received across categories 1 through 4 since 
2002 . 



Table 1 - Applications 


by Research Categories 


identification 


development 


efficacy 


scale-up 


11% 


49% 


31% 


9% 



This distribution has the desirable triangular shape, with identification and development 
representing 60% of applications, small field tests at 31%, and evaluations of scaled-up 
programs at 9%. 

In addition to examining the mix of grant applications across all funding programs, the 
Institute has examined the mix within four critical research programs that have existed 
since 2003. The results from this inventory are presented in Table 2. 



Table 2 - Applications by Researcl 


h Categories by Research Program 




identification 


development 


efficacy 


scale-up 


reading and writing 


5% 


64% 


18% 


13% 


math and science 


10% 


57% 


26% 


7% 


teacher quality 


1% 


68% 


26% 


5% 


policy & management 


27% 


57% 


16% 


0% 



The distribution of grant applications within research programs suggests two issues the 
Institute should address in managing these research programs. First, the teacher quality 
research program is making virtually no use of the identification goal, which typically 
involves statistical analyses of large databases to model and test associations between 
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conditions such as teacher training and outcomes such as student test scores. With the 
growth and availability of state databases that incorporate longitudinal data for students 
and that include individual data on teachers, this is a resource that should be utilized. The 
Institute intends to publicize the need for this type of research, to work with its state 
partners in the statewide longitudinal data systems program, and with its research and 
development center on analysis of longitudinal data to make such research feasible. 

A second issue is the absence of scale-up research within the education policy, 
management, finance, and leadership area. This likely reflects the lack of a substantial 
base of rigorous research in this field. However, evidence suggests that the effectiveness 
of scientifically based programs and practices is strongly influenced by the context of 
management and leadership in the schools and districts in which such programs are 
implemented. To enhance research in this area, the Institute has just launched a new 
research program on education leadership, listed previously in the section on k-12 
education. The Institute also plans new initiatives that directly involve school districts as 
partners in research on policies that are of high relevance to schools districts nationwide. 

Is research yielding findings that will enhance academic achievement? 

As indicated previously, not enough time has passed since the Institute was established for 
the vast majority of its grants to have progressed to the point at which findings can be 
inventoried. An exception is one of the Institute’s first grant programs, on cognition and 
student learning. Grants have been made in this area since 2002, and the initial grants are 
beginning to produce results. This program has been remarkably successful, as indicated 
by its having been the cover story of the March 2006 issue of the Association for 
Psychological Science's Observer (APS Observer - How We Learn) . 

Using educational software that incorporates design principles from cognitive psychology, 
one team of researchers funded by the cognition and student learning program has 
demonstrated four-fold increases in children's learning of science and vocabulary terms 
that are needed for classroom subjects. Another team of researchers has discovered what 
may prove to be a basic principle of instruction — that learning is optimal when instruction 
on content is repeated at from 10 to 20 percent of the interval over which students are 
expected to retain the material. For example, mathematics content presented during the 
first month of a 10-month course should be reviewed in the second month of that course 
for optimal retention on an end-of-course examination. A third team of researchers has 
demonstrated that giving a test after reading prose material produces a greater benefit on a 
final test than many additional readings of the material. 

Findings such as these, which flow from theoretically grounded, methodologically 
rigorous research, are going to transform education if the Institute carries out it priorities 
well. This document lays out a plan to address areas in which insufficient opportunities 
exist for researchers to address priorities and in which the current mix of applications 
indicates gaps in important categories of research. This is an initial plan that has been 
developed through a complete inventory of the Institute’s investments as connected to its 
priorities. The plan is dynamic and will be adjusted annually based on a continuing 
examination of the opportunities, mix, and yield of the Institute’s grant programs. 
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